Learning with options: Just deliberate and relax

Bounded rationality is a very important framework for understanding rationality in both natural and artificial systems. In this paper, we aim to bridge the gap between bounded rationality and reinforcement learning, which has also proven very fruitful in both types of intelligent systems. A lot of reinforcement learning work has focused on Markov Decision Processes, where optimal policies can be obtained under certain assumptions. However, optimality does not take into account possible resource limitations of the agent, which is assumed to have access to a lot of data and computation time. One approach for reducing the costs of such agents has been to provide them with temporally extended actions and models, which allow policies to be computed faster, cf. [Dietterich, 2000; Precup, 2000]. However, the problem of automatically finding good temporal abstractions has proven very difficult. Part of this difficulty stems from the fact that from the point of view of absolute optimality, temporal abstractions are not necessary: the optimal policy is achieved by primitive actions. Therefore, it has been difficult to formalize in what precise theoretical sense temporally abstract actions are helpful.